Higher-Order Predictive Information for Learning an Infinite Stream of Episodes

We consider the problem of lifelong learning from an indefinite stream of temporal episodes, i.e. a time series consisting of episodes, where the number of the episodes is potentially infinite and the length of each episode varies. Examples of this class of learning include a humanoid robot that continually learns to imitate various human behaviors [7], a computer music system that learns to compose from a continuous stream of music pieces [4], and a cognitive system that incrementally learns visual concepts from a series of movies over a long period of time [9]. What kinds of objective function should the lifelong learner use to balance the short-term and long-term performance? How should the learner optimize its model complexity when the statistics of the episodes change over time? Maximization of the expected future reward, such as a value function used in reinforcement learning, might be useful if we could define rewards for a prespecified goal. For learning an indefinite stream of episodes, we find the mutual information-based measures of information theory, such as predictive information [2], I�Xffffff;Xpppf�, and empowerment [3], maxA I(Xf+1;Af|xf) , suitable. The predictive information is, however, typically approximated by restricting the time horizons to a single time step, i.e. I�Xffffff;Xpppf� = I(Xf+1;Xf) . Though this is exact under the Markov assumption, i.e. the probability of a state depends only on the probability of the previous state, and still can generate explorative behavior [1], the predictive power can be improved by increasing the order of temporal dependency. Here we extend the first-order predictive information to the kth-order predictive information for lifelong learning from a continuous stream of time-series episodes. We generalize the predictive information I(Xf+1;Xf) by replacing the first-order term Xf for past by k-th order history term Xf−k:f = Xf−k,f−k+1,...,f, i.e. I(Xf+1;Xf−k:f). The generalized, higher-order predictive information is written as: